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SUMMARY 


Overview 


Four  methods  were  used  to  elicit  judgments  of  relative 
frequency.  Although  formally  equivalent,  these  methods  elicited 
estimates  that  differed  as  much  as  a  hundredfold.  Subsequent 
investigations  revealed  that  careful  attention  must  be  paid  to 
how  people  store  and  process  information  before  one  can  ask  them 
what  they  know  or  try  to  teach  them  more. 

Background 

Decision-making  processes  typically  involve  the  extensive 
interchange  of  information.  Those  involved  state  their  opinions, 
ask  others  for  their  assessments,  and  attend  to  incoming  data. 
Often,  there  are  a  number  of  formally  equivalent  ways  to  express 
any  given  piece  of  information.  Formal  equivalence  is  not, 
however,  a  guarantee  of  psychological  equivalence.  If  different 
modes  of  expression  lead  to  different  answers,  the  result  can 
be  miscommunication  or  ineffective  exploitation  of  existing 
knowledge.  The  present  experiments  studied  the  effect  of  mode 
of  expression  on  the  elicitation  of  information  regarding  the 
risks  of  various  potentially  lethal  events  such  as  automobile 
accidents  and  heart  attacks. 

Approach 

In  two  experiments,  estimates  of  lethality  were  elicited 
in  four  ways:  (1)  direct  estimates  of  death  rate  per  100,000 
afflicted,  (2)  estimates  of  the  number  of  survivors  for  each 
individual  who  succumbs,  (3)  estimates  of  the  number  of  people 


who  died  given  the  number  afflicted,  and  (4)  estimates  of  the 
number  of  survivors  given  the  number  of  casualties.  In  the 
second  experiment,  participants  were  told  the  correct  answers 
after  making  their  estimates;  one  hour  later,  following  a  series 
of  extraneous  tasks,  they  were  unexpectedly  asked  to  remember 
the  correct  answers.  Both  they  and  a  third  group  of  individuals 
were  asked  to  judge  the  "naturalness"  of  the  different  modes 
of  expression. 

Findings  and  Implications 

Judgments  proved  to  be  highly  sensitive  to  mode  of  expres¬ 
sion,  sometimes  varying  by  a  factor  of  one  hundred  for  a  single 
lethal  event.  There  were  corresponding  differences  in  the  accuracy 
of  the  estimates,  which  varied  from  consistent  overestimation  to 
consistent  underestimation,  depending  upon  the  mode  used.  Judges 
showed  some  proficiency  for  learning  the  correct  answers,  with 
the  best  memory  accompanying  the  group  that  initially  showed 
the  worst  performance.  Thus,  the  failure  to  have  information 
available  in  a  particular  way  need  not  indicate  an  inability 
to  utilize  that  mode  of  expression.  The  discussion  considers  the 
importance  of  probing  information-processing  propensities  before 
designing  procedures  for  the  elicitation  of  information. 
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JUDGED  LETHALITY 


One  component  of  the  risk  one  faces  from  any  malady  is 
the  probability  of  being  afflicted  with  it.  A  second  compo¬ 
nent  is  the  probability  of  succumbing,  should  one  be  afflicted. 
The  latter  component,  lethality,  has  proven  in  recent  research 
to  be  a  critical  component  of  people's  attitudes  toward  the 
risks  they  face  in  life.  For  these  attitudes  to  lead  to  appro¬ 
priate  behavior,  people's  lethality  judgments  must  be  reason¬ 
ably  veridical  (Fischhoff,  Slovic  &  Lichtenstein,  in  press; 
Slovic,  Fischhoff  &  Lichtenstein,  1980). 

Table  1  presents  some  evidence  regarding  the  veridicality 
and  impact  of  lethality  judgments.  Lichtenstein,  Slovic,  Fisch¬ 
hoff,  Layman  and  Combs  (1978)  had  people  estimate  the  annual 
number  of  deaths  in  the  U.S.  from  various  causes.  The  causes 
whose  frequency  was  most  over-  or  most  under-estimated  are 
listed,  along  with  lethality  rates  based  on  U.S.  DHEW  statistics 
and  direct  judgments  of  lethality  (on  a  21-point  scale)  from 
Lichtenstein  et  al.  Although  there  appears  to  be  no  relation 
between  bias  in  frequency  estimation  and  these  rough  lethality 
judgments,  the  most  overestimated  causes  are  statistically 
somewhat  more  lethal  than  the  most  underestimated  ones  (p  <  .10; 
Mann-Whitney  test) .  These  lethality  judgments  also  seem  to 
be  reasonably  veridical,  with  a  rank  correlation  of  .70  between 
lethality  judgments  and  statistical  estimates.  Interpretation 
of  this  correlation  should  be  qualified  by  noting  that  the  stat¬ 
istical  values  vary  over  several  orders  of  magnitude.  The 
absence  of  at  least  some  correlation  with  judgments  would  re¬ 
flect  an  extraordinary  degree  of  ignorance.  This  correlation 
could  also  hide  a  great  disparity  between  statistical  and  judged 
values,  a  possibility  that  cannot  be  examined  with  this  (inter- 
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val)  response  mode. 


The  present  study  looks  at  judgments  of  lethality  derived 
by  four,  formally  equivalent,  elicitation  modes,  responses  to 
which  can  be  translated  to  a  form  comparable  with  lethality 
statistics.  It  has  two  foci.  One  is  the  substance  of  lethality 
judgments.  How  accurate  are  they?  Are  there  any  persistent, 
interpretable  biases?  In  particular,  are  there  mis- judgments 
related  to  those  identified  by  Lichtenstein  et  al.  (1978)? 

The  second  focus  is  the  impact  of  elicitation  mode  on  the 
lethality  rates  derived  from  the  responses.  Like  many  other 
numerical  quantities,  lethality  rates  may  be  elicited  in  a 
number  of  formally  equivalent  ways.  For  example,  one  may  ask 
about  the  rate  of  death  or  about  the  rate  of  survival.  One 
may  ask  for  a  direct  rate  estimate  (e.g.,  deaths  per  100,000) 
or  for  estimates  of  the  numbers  of  afflicted  and  dying  which 
can  then  be  converted  into  the  desired  rate.  One  may  or  may 
not  provide  a  context  regarding  the  overall  number  or  rate  of 
deaths.  Conceivably,  these  discretionary  decisions  may  affect 
both  how  people  assess  their  knowledge  and  how  they  translate 
that  knowledge  into  a  response  (Fischhoff,  Slovic  &  Lichtenstein, 
in  press) .  For  example,  if  people  have  difficulty  providing 
large  numerical  responses,  the  smaller  the  correct  answer  in 
the  response  mode  chosen,  the  more  veridical  the  derived  lethality 
rates  will  be.  Or,  a  focus  on  survivors  might  reduce  the  sal¬ 
ience  of  death  and,  thereby,  reduce  judged  lethality  (Tversky 
&  Kahneman,  1973) . 

In  addition  to  shedding  some  light  on  the  storage  and 
evocation  of  risk  information,  the  discovery  of  appreciable 
response-mode  effects  could  have  considerable  applied  implica¬ 
tions.  Those  interested  in  how  much  the  public  knows  about 


Table  1 

Judgments  of  Lethality  and  Bias  in  Frequency  Judgments 


Cause  of  Death 

Judged  a 
Lethality 

Statistical 

Lethality 

Frequency  .lost 

Overestimated 

1. 

All  Accidents 

7.0 

.  03 

2. 

Motor  Vehicle  Accidents 

9.0 

.025 

3. 

Pregnancy,  Childbirth,  Abortion 

4.6 

.01 

4. 

Tornado 

6.3 

— 

5. 

Flood 

6.5 

— 

6. 

Botulism 

10 . 3 

.5 

7. 

All  Cancer 

13.2 

.372 

8. 

Fire  and  Flames 

10.6 

— 

9. 

Venomous  Bite  or  Sting 

6.8 

— 

10. 

Homicide 

18.3 

1.0 

Frequency  Most 

Underestimated 

1. 

Smallpox  Vaccination 

0.7 

.000 

2. 

Diabetes 

6.5 

.008 

3. 

Stomach  Cancer 

11.9 

— 

4. 

Lightning 

10 . 1 

— 

5 . 

Stroke 

11.8 

.116 

6  . 

Tuberculosis 

7.7 

.018 

7. 

As  thma 

2.1 

.0003 

8. 

Emphysema 

11.0 

.014 

aJudgments  of  lethality  with  0  =  never  fatal;  20  =  certainly  fatal. 
Probability  (death | affliction) .  Blank  indicates  not  available. 


lethality  could  derive  quite  different  appraisals  of  the  public' 
knowledge  or  ignorance  from  surveys  of  knowledge  conducted  with 
different  methods.  Those  appraisals  could,  in  turn,  affect  the 
credibility  afforded  public  concerns  about  risk  in  the  manage¬ 
ment  of  hazards.  Where  public  education  is  the  order  of  the 
day,  information  should,  presumably,  be  presented  in  the  mode 
most  compatible  with  the  recipients'  "natural"  representation 
of  such  knowledge. 


Experiment  1 

Method 

Stimuli .  Twenty  causes  of  death  were  chosen  on  the  basis 
of  their  familiarity  to  subjects,  the  availability  of  statis¬ 
tics  on  their  incidence,  and  the  stability  of  lethality  rates 
over  recent  years.  These  statistics  were  derived  from  U.S. 
public  health  statistics,  many  of  which  were  collected  in 
Chemical  and  Engineering  News  for  December  5,  1977.  They 
appear  in  the  right-hand  column  of  Table  2,  and  represent 
100,000  times  the  ratio  of  the  number  of  people  who  die  from 
a  cause  each  year  to  the  total  number  who  become  afflicted 
with  it  during  the  year. 

Design.  Subjects  were  asked  either  to  estimate  lethality 
rates  directly  or  to  provide  numerical  estimates  that  could 
be  readily  converted  to  lethality  rates.  Both  the  "rate"  and 
the  "number"  questions  were  posed  in  two  ways,  one  focusing 
on  deaths  and  the  other  focusing  on  survivors. 

Instructions .  All  questionnaires  began  with  the  following 
general  information: 

There  are  rougly  215,000,000  people  in  the  United  States 

Some  1,900,000  die  each  year  ....  Of  these,  about 
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1,750,000  succumb  to  diseases;  100,000  die  due  to  accidents 

while  50,000  either  commit  suicide  or  are  murdered. 

In  this  questionnaire,  we  would  like  you  to  estimate 

the  lethality  of  each  of  a  number  of  problems. 

A  parenthetical  remark  inserted  at  the  ellipses  in  the  _irst 
paragraph  translated  these  overall  statistics  into  terms  con¬ 
sistent  with  the  response  mode  each  group  used  (e.g.,  "about 
884  per  100,000  people"  for  the  estimate  death  rate  group 
described  immediately  below) . 

Exemplary  questions  from  each  of  the  four  conditions  follow 

(a)  Estimate  death  rate:  In  a  normal  year,  for  each 

100,000  people  who  have  influenza,  how  many  people  do  you  think 
die  of  influenza? _ 

(b)  Estimate  number  died:  Last  year,  80,000,000  people 

had  influenza.  How  many  of  them  do  you  think  died  of  it? _ 

(c)  Estimate  survival  rate:  In  a  normal  year,  for  each 

person  who  dies  of  influenza,  how  many  do  you  think  have  influ¬ 
enza  but  do  not  die  of  it  during  the  year? _ 

(d)  Estimate  number  survived:  In  a  normal  year,  5,000 

people  die  of  influenza.  How  many  people  do  you  think  have 
influenza,  but  do  not  die  from  it  during  the  year? _ 

Thus,  the  death  rate  and  survival  rate  conditions  called 
for  estimates  of  rates,  whereas  the  number  died  and  number 
survived  conditions  called  for  estimates  of  numbers  of  people. 
The  death  rate  and  number  died  conditions  dealt  with  fatali¬ 
ties,  whereas  the  survival  rate  and  number  survived  conditions 
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dealt  with  survivors.  Minor  wording  changes  were  used  with 
some  maladies:  for  drug  abuse,  "have"  was  replaced  with 
"suffer  from;"  "are  injured"  was  used  for  automobile  accidents; 
"become"  was  used  for  pregnancy. 

Subjects .  One  hundred  and  fifty-eight  individuals  took 
part;  40  in  the  death  rate  group,  38  in  the  number  died  group, 
40  in  the  survival  rate  group,  and  40  in  the  number  survived 
group . 

Results 


Subject  culling.  One  subject  in  the  death  rate  group  and 
five  in  the  survival  rate  group  responded  with  percentages 
rather  than  with  numbers  and  were  eliminated. 

Data  analysis.  In  order  to  facilitate  comparisons  between 
groups,  responses  were  translated  into  death  rates  per  100,000 
individuals  afflicted.  The  death  rate  group  estimated  these 
rates  directly.  For  the  number  died  group,  which  was  given 
the  number  afflicted  and  asked  to  estimate  the  number  of  fatal¬ 
ities,  the  appropriate  ratio  was  calculated  and  multiplied  by 
100,000.  The  survival  rate  group '  s  estimates  of  the  number  of 
survivors  per  fatality  were  confetted  to  a  death  rate  by  multi¬ 
plying  the  reciprocal  of  [1+  estimated  number  of  survivors] 
by  100,000.  The  final  group's  estimates  of  the  number  of  sur¬ 
vivors  were  converted  to  a  death  rate  by  dividing  the  number 
of  dead  (given  to  subjects)  by  the  sum  of  the  number  of  dead 
and  the  estimated  number  of  survivors  (and  multiplying  by  100,000) 
Individual  subjects'  converted  responses  were  summarized  by 
geometric,  rather  than  arithmetic,  means  so  as  to  reduce  the 
influence  of  occasional  outliers.  They  appear  in  Table  2,  the 
bottom  row  of  which  presents  coefficients  of  concordance  for 
each  group.  That  coefficient  represents  the  mean  of  the  corre- 
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lations  between  each  pair  of  subjects'  rankings  of  the 
maladies  by  lethality.  There  was  fairly  high  agreement 
within  the  death  rate,  number  died,  and  number  survived 
groups;  rather  low  agreement  within  the  survival  rate  group. 

Agreement  between  response  modes.  Table  2  presents  the 
geometric  means  of  the  derived  death  rates.  The  four  columns 
differ  markedly  in  the  magnitude  of  the  death  rates  they  include. 
These  differences  provide  an  ordering  of  the  response  modes 
by  the  magnitude  of  the  estimates  they  produce.  Number  sur¬ 
vived  estimates  are  greater  than  those  for  death  rate  for  75% 
of  the  items;  death  rate  estimates  are  greater  than  number 
died  estimates  for  75%  of  the  items;  whereas  75%  of  the  number 
died  estimates  are  greater  than  the  corresponding  survival  rate 
estimates.  Since  instructions  to  estimate  the  number  of  sur¬ 
vivors  produced  the  largest  death  rates  and  instructions  to 
estimate  the  survival  rate  produced  the  lowest  rates,  the 
substantive  focus  of  the  task  (survivors  or  dead)  cannot  be 
the  sole  determinant  of  the  magnitude  of  estimated  death  rates. 

In  terms  of  magnitude,  the  statistical  death  rate  fell  in  the 
middle  of  the  four  sets  of  estimated  rates.  Thus,  whether  these 
individuals  tended  to  over-  or  underestimate  lethality  depends 
upon  how  the  question  was  asked.  Number  survived  and  death 
rate  instructions  produce  overestimates  in  £3%  and  55%  of  the 
cases,  respectively,  whereas  number  died’ and  survival  rate 
produced  underestimates  in  53%  and  85%  of  the  cases,  respec¬ 
tively  . 

Despite  these  discrepancies  in  absolute  estimates,  there 
was  general  agreement  regarding  the  relative  lethality  of  these 
20  maladies.  Rank  correlations  between  the  entries  in  Table  2 
were  all  in  the  range  .72  to  .83  and  were  all  significant 
statistically  (p  <  .001).  There  was  no  tendency  for  estimates 
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Table  2 

Direct  and  Converted  Lethality  Rate  Estimates 
Based  on  Geometric  Mean  Responses 


Death  Rate 

per  100, 

000  Afflicted 

Malady 

Estimate 
Death 
Rate  a 

Estimate 

Number 

Died 

Estimate 

Survival 

Rate 

Estimate 

Number 

Survived 

Statistical 

Death 

Rate 

Dental  Problems 

10 

1 

2 

1 

1 

Influenza 

393 

6 

26 

511 

6 

Mumps 

44 

114 

19 

4 

12 

Skin  Diseases 

63 

4 

6 

641 

30 

As  thma 

155 

12 

14 

599 

33 

Alcoholism 

559 

70 

13 

294 

44 

Venereal  Disease 

91 

63 

8 

111 

50 

Measles 

52 

187 

18 

28 

75 

High  Blood  Pressure 

535 

89 

17 

538 

76 

Drug  Abuse 

1,020 

1,371 

19 

95 

80 

Bronchitis 

162 

19 

43 

2,111 

85 

Pregnancy 

67 

24 

13 

787 

250 

Diabetes 

487 

101 

52 

5,666 

800 

Emphysema 

1,153 

1,998 

70 

5,417 

1,423 

Tuberculosis 

S52 

1,783 

188 

8,520 

1,535 

Pneumonia 

563 

304 

77 

9,553 

1,733 

Automobile  Accidents 

6,195 

3,272 

31 

6,813 

2,500 

Strokes 

11,011 

4,648 

181 

24,758 

11,765 

Heart  Attacks 

13,011 

3,666 

131 

27,477 

16,250 

Cancer 

10,889 

10,475 

160 

21,749 

37,500 

Coefficient  of  Concor-  .52 
dance 

.  67 

.34 

r- 

VO 

aOnly  these  rates  were  estimated  directly.  Participants  in  other  groups 
estimated  other  quantities,  which  were  converted  to  lethality  rates  as 
described  in  the  text. 
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with  a  common  focus  (death  or  survival)  to  be  more  highly 
correlated  than  those  without,  and  a  slight  tendency  for  those 
using  the  same  mode  (number  or  rate)  to  be  more  poorly  corre¬ 
lated  (mean  correlation  =  .75  vs.  .81),  although  the  small 
number  of  comparison  groups  makes  these  trends  untrustworthy. 
Using  the  geometric  means  of  estimates  of  the  20  maladies,  the 
highest  correlation  (.83)  was  between  the  death  rate 
and  number  survived  groups,  which  differed  in  both 
focus  and  mode.  Despite  the  relatively  low  agreement  among 
subjects  within  the  survival  rate  group,  their  aggregate  esti¬ 
mates  agreed  reasonably  well  with  those  of  the  other  groups 
(mean  correlation  =  .80). 

Validity  of  Death  Rate  Estimates 

In  a  correlational  sense,  all  judgmental  estimates  were 
highly  related  to  the  statistical  estimates,  with  rank  corre¬ 
lations  ranging  between  .82  (survival  rate)  and  .86  (number 
survived) .  As  Table  2  shows,  however,  these  high  correlations 
obscure  substantial  differences  in  the  actual  estimates  and 
their  degree  of  accuracy.  One  measure  of  the  absolute  accuracy 
estimates  might  be  called  the  error  ratio,  created  by  dividing 
the  estimated  rate  by  the  statistical  rate  for  a  particular 
malady.  Table  3  shows  the  median  of  these  ratios  for  each  group, 
calculated  on  the  20  items  means  appearing  in  Table  2.  As  the 
ordering  of  the  groups  by  magnitude  would  lead  one  to  expect, 
the  death  rate  and  numbers  survived  groups  produced  the  greatest 
ratios,  representing  overestimation,  whereas  the  survival  rate 
group  produced  the  smallest  ratios,  reflecting  underestimation. 
Estimating  the  number  of  deaths  produced  as  many  cases  of  over¬ 
estimation  as  of  underestimation.  The  absence  of  any  systematic 
error  in  this  group  need  not,  however,  guarantee  that  its  esti¬ 
mates  were  the  most  accurate.  Perhaps  a  more  telling  measure 
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of  overall  accuracy  is  derived  by  ignoring  the  direction 
of  the  error.  This  "error  factor"  is  equal  to  the  error  ratio 
or  its  reciprocal,  whichever  is  larger.  As  Table  3  shows, 
the  number  died  group  was  also  most  accurate  in  this  sense, 
although  it  was  not  greatly  superior  to  the  death  rate  and 
number  survived  groups .  Survival  rate  estimates  are  substan¬ 
tially  inferior.  The  bottom  line  of  Table  3  presents  the  geo¬ 
metric  means  of  the  error  factors  for  each  individual  estimate. 
Since  they  do  not  allow  overestimates  and  underestimates  for 
a  particular  malady  to  cancel  one  another  out,  these  figures 
are  quite  a  bit  larger  than  the  error  factors  (in  the  row  above) , 
obtained  by  first  calculating  the  geometric  mean  response  for 
each  malady.  Nonetheless,  they  show  the  same  pattern.  Indi¬ 
vidual  estimates  in  the  death  rate,  number  died,  and  number 
survived  groups  are  off  by  about  the  same  amount  (about  a  factor 
of  10) ,  whereas  survival  rate  estimates  are  considerably  more 
inaccurate,  being  off  by  a  factor  of  about  30. 

Particular  causes  of  death.  Table  4  presents  causes  of 
death  whose  death  rate  was  systematically  over-  or  underesti¬ 
mated  relative  to  the  median  error  ratio  for  the  various  groups. 
Thus,  for  example,  the  dental  problems  death  rate  was  overesti¬ 
mated  by  the  estimate  death  rate  group  since  its  error  ratio 
(10)  was  larger  than  the  median  error  ratio  (1.86).  In  general, 
the  highest  death  rates  were  consistently  underestimated  and 
the  lowest  overestimated.  The  magnitude  of  this  flattening 
may  be  crudely  assessed  by  the  slope  of  the  regression  line 
predicting  log  estimated  rate  from  log  statistical  rate  for  the 
number  died  group,  the  group  without  any  overall  tendency  to 
over-  or  underestimate.  The  slope  equals  .78  (with  r  =  .87). 

The  most  marked  exceptions  to  this  tendency  are  the  underesti¬ 
mation  of  death  rates  from  pregnancy  and  venereal  disease  and 
the  overestimation  of  death  rates  from  bronchitis  and  high 
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Table  3 

Magnitude  of  Error 
Experiment  1 


Estimate 

Death 

Rate 

Estimate 

Number 

Died 

Estimate 

Survival 

Rate 

Estimate 

Number 

Survived 

Median  of  Item 

Error  Ratios3 

1.86 

1.08 

.18 

3.48 

Median  of  Item 

Error  Factors0 

3.26 

2.64 

5.63 

3.48 

Geometric  Mean  of 

Error  Factors  for 
Individual  Estimates 

10.9 

10.2 

33.2 

12.5 

aError  ratio  =  estimated  lethality  -r  statistical  lethality 
^Error  factor  =  larger  of  (error  ratio  or  its  reciprocal) 


Table  4 

Persistent  Biases  in  Judged  Death  Rates 
Relative  to  Median  Error  Ratio  in  Each  Group 
Experiment  1 


Rank  of  Statistical 

Cause  of  Death 

Lethality  Rate 

Always  Underestimated 


Cancer 

20 

Heart  Attack 

19 

Stroke 

18 

Pregnancy 

12 

Underestimated  in  3  of  4  Methods 


16 

13 

7 


Overestimated  in  3  of  4  Methods 


Pneumonia 

Diabetes 

Venereal  Disease 


Always  Overestimated 


High  Blood  Pressure  9 

Alcoholism  6 

Influenza  2 

Note:  Measles  (8),  tuberculosis  (15),  auto  accidents  (17), 
emphysema  (14)  were  overestimated  by  two  groups  and  underestimated 
by  two  groups.  Rank  in  parentheses. 


blood  pressure.  These  might  be  viewed  as  cases  of  misinfor¬ 
mation  above  and  beyond  any  tendency  subjects  have  to  underesti¬ 
mate  the  difference  in  the  lethality  rates  of  the  most  and 
least  lethal  maladies. 

Extending  this  reasoning,  one  might  assume  that  there  is 
not  only  a  general  bias  toward  over-  or  underestimation  in  each 
group,  but  also  a  systematic  "primary  bias"  toward  flattening 
the  response  scale.  Deviations  from  the  regression  lines  pre¬ 
dicting  judgmental  estimates  from  statistical  estimates  could 
be  called  "secondary  biases,"  following  Lichtenstein  et  al. 
(1978).  Table  5  shows  the  direction  and  rough  magnitude  of 
these  secondary  biases  in  the  four  conditions.  In  this  light, 
some  of  the  most  lethal  events,  in  terms  of  both  their  statis¬ 
tical  lethality  rate  and  total  number  of  people  killed  (cancer, 
strokes,  heart  attacks,  emphysema,  high  blood  pressure)  were 
rather  accurately  judged  in  this  perspective.  Skin  diseases, 
venereal  disease,  and  dental  problems,  which  afflict  many  but 
kill  very  few,  were  overestimated.  On  the  other  hand,  influ¬ 
enza,  which  kills  many  people  in  terms  of  absolute  numbers, 
but  also  afflicts  a  large  portion  of  the  population  in  any 
given  year,  was  overestimated.  The  mixed  pattern  of  secondary 
biases  for  the  other  maladies  lends  itself  to  no  simple  sum¬ 
mary.  Influenza,  skin  disease,  dental  problems,  and  pregnancy 
were  among  the  few  causes  of  death  to  assume  the  same  status 
in  Tables  4  and  5,  indicating  that  any  discussion  of  relative 
bias  in  estimates  for  particular  maladies  depends  upon  whether 
one  treats  the  flattening  observed  here  and  elsewhere  (Lichten¬ 
stein  et  al.  ,  1978;  Slovic,  Fischhoff  &  Lichtenstein,  1979) 
as  a  substantive  result  or  a  statistical  artifact. 

Discussion 


Four  methods  were  used  to  obtain  estimates  of  the  lethality 
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Table  5 

Direction  of  Secondary  Bias 
Experiment  1 


Malady 

Estimate 

Death 

Rate 

Estimate 

Number 

Died 

Estimate 

Survival 

Rate 

Estimate 

Number 

Survived 

Dental  Problems 

- 

— 

— 

_ 

Influenza 

++ 

+ 

++ 

++ 

Mumps 

- 

++ 

++ 

— 

Skin  Diseases 

— 

— 

— 

+ 

As  thma 

0 

— 

- 

+ 

Alcoholism 

++ 

0 

0 

+ 

Venereal  Disease 

— 

0 

— 

- 

Measles 

— 

+ 

0 

— 

High  Blood  Pressure 

+ 

0 

0 

0 

Drug  Abuse 

++ 

+  + 

0 

— 

Bronchitis 

- 

— 

++ 

++ 

Pregnancy 

— 

— 

— 

0 

Diabetes 

- 

— 

0 

+ 

Emphysema 

0 

+ 

+ 

0 

Tuberculosis 

- 

+ 

++ 

+ 

Pneumonia 

— 

— 

+ 

+ 

Auto  Accidents 

++ 

+ 

— 

0 

Strokes 

++ 

0 

0 

0 

Heart  Attacks 

++ 

0 

0 

0 

Cancer 

0 

0 

0 

- 

++  Strongly  overestimated 
+  Somewhat  overestimated 
0  No  appreciable  secondary  bias 
Somewhat  underestimated 
—  Strongly  underestimated 
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of  20  maladies.  The  estimated  death  rates  derived  from  sub¬ 
jects'  responses  to  the  four  methods  showed  similar  rank 
orderings.  These  were,  in  turn,  similar  to  the  ranking  of 
available  statistical  estimates.  In  this  sense,  subjects 
exhibited  an  overall  knowledge  of  lethality  rates  similar 
to  the  knowledge  of  the  frequency  of  death  from  various  causes 
exhibited  by  the  subjects  in  Lichtenstein  et  al.  (1978) ,  who 
also  found  similar  orderings  with  different  response  modes. 

Nonetheless,  the  estimates  obtained  with  the  different 
methods  varied  greatly  in  their  absolute  values,  ranging 
from  consistent  overestimation  of  the  statistical  rate  with 
the  number  survived  group  to  consistent  underestimation  with 
the  survival  rate  group.  Thus,  the  estimated  lethality  rate 
for  diabetes,  for  example,  assumed  values  from  7  times  too 
large  to  15  times  too  small,  depending  upon  the  response  mode 
chosen.  It  is  possible  to  identify  particular  maladies  that 
are  consistently  over-  or  underestimated  relative  to  the  "over¬ 
all  biases"  induced  by  the  different  methods.  Such  "item 
biases"  were  also  observed  by  Lichtenstein  et  al.  (1978). 

Of  course,  one  need  not  take  the  statistical  lethality 
rate  as  an  absolute  standard.  Public  health  statistics,  like 
all  other  statistics,  are  fallible.  Poor  sampling  and  incom¬ 
plete  reporting  are  two  obvious  problems;  uncertainty  about 
how  to  attribute  causes  of  death  is  another.  For  example,  if 
high  blood  pressure  predisposes  someone  to  heart  attacks,  or 
if  cancer  weakens  resistance  to  tuberculosis,  or  if  alcoholism 
encourages  reckless  driving,  which  cause  is  to  be  implicated? 
Even  where  a  standard  attribution  has  been  adopted,  it  may  not 
be  known  and  utilized  by  all  the  physicians  who  produce  these 
statistics.  As  a  result,  some  of  respondents'  estimates  may 
be  more  or  less  accurate  than  suggested  in  these  tables.  Such 
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errors  would  not,  however,  explain  the  discrepancies  between 
the  rates  obtained  with  different  response  modes. 


How  might  those  discrepancies  be  explained?  One  possibil¬ 
ity  is  to  look  for  substantive  differences  between  the  tasks. 
Although  all  four  tasks  consider  the  same  elements,  two  of 
them  present  additional  information  in  the  course  of  posing 
their  questions :  Number  died  instructions  tell  how  many  people 
were  afflicted;  number  survived  instructions  tell  how  many  died 
That  information  was  designed  to  equate  the  difficulty  of  their 
tasks  with  that  of  the  other  tasks.  For  respondents  who  knew 
the  lethality  rate,  its  provision  would  have  improved  perfor¬ 
mance.  For  respondents  whose  lethality  estimates  were  in  error 
it  might  have  made  matters  better  or  worse  by  introducing  com¬ 
pensating  or  exaggerating  biases.  A  more  interesting  possibil¬ 
ity  is  that  the  information  changed  how  respondents  thought 
about  the  problem,  or  even  how  they  defined  the  maladies.  For 
example,  being  told  that  80  million  people  were  afflicted  with 
influenza  may  have  surprised  the  number  died  group  and  encour¬ 
aged  them  to  think  of  it  as  "common  flu,"  a  non- threatening 
nuisance  that  merited  a  low  lethality  estimate.  Other  specu¬ 
lative  examples  are  also  possible,  particularly  when  one  con¬ 
siders  the  ambiguous  nature  of  affliction  with  some  maladies 
(e.g.,  drug  abuse,  alcoholism).  Such  explanations  would  re¬ 
quire  a  substantive  theory  for  each  malady,  regarding  what 
people  know  already  and  what  they  learn  or  infer  from  the  ques¬ 
tion;  these  explanations  might  also  be  quite  sensitive  to  the 
nature  (or  knowledge)  of  the  subject  population  studied,  at 
least  more  so  than  explanations  involving  more  fundamental  cog¬ 
nitive  processes. 


Whenever  one  deals  with  such  "rich"  stimuli  like  the  pre¬ 
sent  ones,  speculative,  item-specific  explanations  are  possible 
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Aside  from  being  unparsimonious ,  unsupported,  and  unsatis¬ 
fying,  such  explanations  cannot  account  for  the  systematic 
differences  between  the  lethality  estimates  of  the  four 
groups.  Nor  can  they  cope  with  the  consistent,  order-of- 
magnitude  differences  between  the  death  rate  and  survival 
rate  groups,  which  received  no  additional  information.  Another, 
more  artifactual,  mode  of  explanation  is  to  look  at  the  size 
of  the  responses  each  group  was  required  to  provide,  under  the 
hypothesis  that  people  may  have  difficulty  giving  very  large 
or  very  small  (fractional)  responses  (Poulton,  1968). 

Table  6  presents  the  correct  answer  for  each  condition,  the 
number  that,  when  converted,  would  produce  the  statistical 
lethality  rate.  The  number  survived  group  was  required  to  pro¬ 
duce  the  largest  numbers.  Inability  to  do  so  would  mean  under¬ 
estimating  the  number  of  survivors  and  emerge  as  overestimation 
of  the  lethality  rate,  the  result  obtained.  This  particular 
response-bias  explanation  (failure  to  give  sufficiently  large 
numbers)  does  not,  however,  hold  up  for  the  remaining  groups. 
Although  each  group  was  required  to  produce  numbers  in  a  simi¬ 
lar  range,  they  showed  quite  different  systematic  biases.  The 
death  rate  group  produced  overestimates;  the  number  survived 
group  gave  numbers  which  were  too  large  and  which  emerged  as 
unduly  small  lethality  rates,  whereas  there  was  no  systematic 
bias  in  the  number  died  group.'" 

A  third  explanatory  strategy  is  to  ask  whether  the  differ¬ 
ent  tasks  did  not  prime  the  availability  of  certain  considera¬ 
tions  and  thereby  increase  their  judged  likelihood  (Tversky  & 
Kahneman,  1973).  All  other  things  being  equal,  reliance  on 
the  availability  heuristic  should  lead  to  higher  estimates 
with  tasks  focusing  on  death  than  with  those  focusing  on  sur¬ 
vival.  Althc  .gh  the  number  died  and  death  rate  groups  did  have 
similar  error  ratios  (indicating  the  overall  level  of  their 
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Table  6 

Correct  Answers 


Malady 

Estimate 

Death 

Ratea 

Estimate 

Number 

Died 

Estimate 

Survival 

Rate 

Estimate 

Number 

Survived 

Dental  Problems 

1 

40 

162,500 

6,500,000 

Influenza 

6 

5,000 

16,000 

80,000,000 

Mumps 

12 

15 

8,000 

120,000 

Skin  Disease 

30 

1,800 

3,333 

6,000,000 

As  thma 

33 

2,000 

3,000 

6,000,000 

Alcoholism 

44 

4,000 

2,250 

9,000,000 

Venereal  Disease 

50 

500 

2,000 

1,000,000 

Measles 

75 

90 

1,333 

120 ,000 

High  Blood  Pressure 

76 

17 , 500 

1,314 

23,000,000 

Drug  Abuse 

80 

400 

1,250 

500,000 

Bronchitis 

85 

5,500 

1,182 

6,500,000 

Pregnancy 

250 

15,000 

400 

6,000,000 

Diabetes 

800 

36,000 

124 

4,460,000 

Emphysema 

1,423 

18,500 

69 

1,280,000 

Tuberculosis 

1,535 

3 , 300 

64 

212 ,000 

Pneumonia 

1,733 

52,000 

58 

3,000,000 

Auto  Accident 

2,500 

50,000 

39 

1,950,000 

Stroke 

11,765 

200,000 

m 

r- 

1,500,000 

Heart  Attack 

16 , 250 

650 , 000 

5 

3,350,000 

Cancer 

37,500 

375,000 

1.8 

675,000 

aThis  is  also  the  statistical  estimate. 


responses),  they  were  bracketed  by  the  two  "survivor"  groups. 

The  survival  rate  group  had  by  far  the  lowest  estimates,  whereas 
the  number  survived  group  had  the  highest  ratio. 

Although  inherently  unsatisfying,  hybrid  explanations  are 
also  possible.  A  tentative  account  of  these  systematic  biases 
might  be  that  the  death  rate  group  overestimated  lethality 
because  they  were  forced  to  think  of  deaths,  whereas  the  sur¬ 
vival  rate  group  underestimated  lethality  because  of  being 
forced  to  think  of  survivors.  The  same  did  not  happen  with 
the  other  "survivors"  group,  because  the  required  responses 
(the  number  of  survivors)  were  just  too  large.  The  relative 
success  of  the  number  died  group  might  be  attributed  to  its 
response  mode  being  the  most  "natural"  way  to  think  about  these 
matters . 

This  last  set  of  arguments  assumes,  of  course,  that  people 
do  have  basically  accurate  perceptions  of  lethality;  the  more 
compatible  a  response  mode  is  to  the  natural  way  in  which  those 
perceptions  are  organized,  the  truer  the  translation  of  those 
perceptions  into  expressed  judgments.  On  the  other  hand,  if 
one  believed  that  people  tend  to  overestimate  or  underestimate 
lethality,  comparable  cases  might  be  made  for  the  naturalness 
of  the  number  survived  or  survival  rate  perspectives,  respec¬ 
tively.  With  proper  assumptions  about  what  people  know,  a 
claim  could  even  be  made  for  the  naturalness  of  the  survival 
rate  perspective,  although  in  doing  so,  one  would  have  to 
contend  with  the  low  agreement  among  those  subjects  and  their 
poor  discrimination  between  maladies  with  greatly  varying  le¬ 
thality  rates.  Experiment  2  attempts  to  study  naturalness 
with  two  straightforward  strategies.  One  is  asking  for 
explicit  judgments  of  naturalness,  bearing  in  mind  that  such 
assessments  are  only  as  good  as  the  quality  of  the  introspec- 


19 


tions  upon  which  they  rely  (Ericsson  &  Simon,  1980)  .  The 
second  involves  looking  at  people's  ability  to  utilize  informa¬ 
tion  presented  in  each  of  the  modes.  In  Experiment  2  as  in 
Experiment  1,  subjects  first  made  estimates  in  one  of  the  four 
modes.  They  then  receive  the  statistically  correct  answer  (in 
that  mode).  After  a  series  of  intervening  tasks,  they  are  un¬ 
expectedly  asked  to  remember  that  correct  answer.  Arguably, 
the  best  recall  and  the  greatest  improvement  in  knowledge  will 
indicate  the  most  natural  representation,  that  mode  most  con¬ 
ducive  to  the  integration  and  preservation  of  additional  know¬ 
ledge. 

Experiment  2 

Method 

Stimuli .  Ten  of  the  twenty  maladies  used  in  Experiment  1 
were  selected  for  Experiment  2  by  eliminating  those  ten  for 
which  public  health  statistics  seemed  most  unsatisfactory. 

The  remaining  ones  allowed  reasonable  confidence  in  the  diag¬ 
nosis  of  affliction  and  the  attribution  of  responsibility  for 
death.  They  can  be  found  listed  in  various  tables  of  results 
below. 

Design.  Participants'  first  task  was  to  make  estimates 
following  the  procedures  for  one  of  the  four  response  modes 
of  Experiment  1.  Immediately  afterward,  they  were  given  the 
statistical  answer  as  the  "true  value."  To  encourage  them  to 
attend  to  this  value,  they  were  asked  to  score  their  own 
answers  as  being  higher  or  lower  than  the  true  value.  Approxi¬ 
mately  one  hour  later,  after  a  series  of  unrelated  tasks, 
respondents  were  asked  to  recall  the  true  value  they  had  been 
given.  Finally,  they  were  given  statistical  lethality  estimates 


for  a  new  malady,  infectious  hepatitis,  phrased  in  formats  con¬ 
sistent  with  each  of  the  four  response  modes.  Their  task  was 
to  rank  those  phrasings  according  to  how  "natural"  they  seemed. 
A  final  group  received  just  the  one  page  asking  for  naturalness 
rankings  without  any  preceding  judgment  tasks. 

Instructions .  The  initial  estimation  task  was  identical 
to  that  of  Experiment  1.  The  subsequent  scoring  task  gave 
the  true  value  rounded  to  two  significant  figures  and  told 
subjects : 

For  each  problem,  compare  your  estimates  from  Part  I 
with  the  true  value.  If  your  estimate  for  a  particular 
problem  was  too  high,  place  a  check  in  the  column  labeled 
"my  estimate  too  high."  If  your  estimate  was  too  low, 
place  a  check  in  the  column  labeled  "my  estimate  too  low." 
If  your  estimate  was  exactly  correct,  leave  the  space 
blank. 

In  the  recall  task,  the  following  instructions  were  used 
to  help  ensure  that  respondents  gave  their  recollection  of  the 
true  value,  rather  than  their  own  original  response. 

In  an  earlier  part  of  this  experiment,  you  made  esti¬ 
mates  of  the  lethality  of  a  number  of  problems.  After 
you  made  your  estimates,  you  compared  each  of  them  to  the 
true  value  and  indicated  for  each  estimate  whether  you 
were  above  or  below  the  true  value. 

Listed  below  are  each  of  the  problems  you  saw  before. 
Now,  we  would  like  you  to  recall  as  best  you  can  the 
true  values  for  each  of  the  problems.  In  the  space  next 
to  each  problem,  write  your  recollection  of  the  true  value. 

Remember,  it  is  the  true  value  we  would  like  you  to 
do  your  best  to  recall,  not  your  estimate.  If  you  feel 
unable  to  recall  a  true  value  exactly,  make  a  best  guess. 
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The  final  task  was  introduced  as  follows: 


The  lethality  of  infectious  hepatitis  is  phrased  below 
in  four  different  ways.  Read  each  of  the  four  phrasings 
carefully  and  decide  which  expression  seems  most  natural 
to  you.  That  is,  which  phrasing  describes  the  lethality 
of  infectious  hepatitis  in  terms  that  correspond  most 
closely  to  the  way  you  usually  think  of  the  lethality 
of  diseases  and  accidents? 

Subjects .  Two  hundred  and  thirty- four  individuals  took 
part:  37  in  the  death  rate  group,  36  in  the  number  died  group, 

37  in  the  survival  rate  group,  36  in  the  number  survived  group, 
and  87  in  the  group  that  only  rated  naturalness. 

Results 


Subject  culling.  Seven  subjects  used  the  wrong  response 
mode  and  were  eliminated.  No  more  than  two  came  from  any  one 
group. 

Initial  estimates.  Table  7  presents  the  geometric  means 
of  subjects'  initial  estimates.  In  general,  these  estimates 
resemble  the  comparable  estimates  from  Experiment  1  (presented 
in  Table  2).  Across  the  four  groups,  26  of  the  40  present  esti¬ 
mates  are  within  a  factor  of  2  of  the  comparable  Experiment  1 
estimates;  all  40  are  within  a  factor  of  5.  Again,  the  coeffi¬ 
cients  of  concordance  reflected  considerable  agreement  among 
subjects  within  all  but  the  survival  rate  group.  As  would  be 
expected,  given  this  general  pattern  of  similarity,  particular 
maladies  that  tended  to  be  relatively  over-  or  underestimated 
in  Experiment  1  (as  presented  in  Tables  4  and  5)  generally 
assumed  the  same  status  here.  Perhaps  the  only  changes  of  note 
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Initial  and  Recalled  Lethality  Rates 
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from  Experiment  1  were  that  the  present  death  rate  group  gave 
somewhat  lower  estimates,  although  there  was  no  great  change 
in  their  accuracy,  and  the  survival  rate  group  gave  somewhat 
higher  estimates,  thereby  moderating  their  tendency  to  under¬ 
estimate  in  Experiment  1.  It  is  unclear  whether  this  modest 
improvement  reflects  greater  knowledgeability  of  the  present 
subjects  or  some  change  due  to  deleting  10  maladies  (e.g., 
helping  subjects  to  concentrate  on  the  remaining  ten) . 

Perhaps  the  most  important  substantive  result  of  Experi¬ 
ment  1  to  be  replicated  was  that  although  the  ranking  of  the 
items  was  similar  in  the  different  groups  (intergroup  rank 
correlations  between  .56  and  .77),  the  absolute  value  of  the 
lethality  rates  varied  systematically  across  groups.  Again, 
the  groups  fell  into  the  following  order  according  to  the  size 
of  their  estimates:  number  survived,  death  rate,  number  died, 
survival  rate  (although  the  difference  between  the  middle  two 
was  minimal).  In  terms  of  order  of  magnitude,  the  statistical 
rate  came  after  number  survived.  Thus,  one  could  again  find 
tendencies  toward  overestimation  or  underestimation,  depending 
upon  the  group  whose  estimates  were  being  examined.  Table  8 
presents  the  error  ratios  and  error  factors  for  Experiment  2, 
along  with  corresponding  results  for  this  subset  of  10  items 
from  Experiment  1.  Except  for  the  modest  changes  in  the  magni¬ 
tude  of  the  death  rate  and  survival  rate  group's  estimates  noted 
above,  the  pattern  here  is  quite  the  same.  The  number  died 
group's  estimates  were  most  accurate,  followed  by  death  rate, 
number  survived,  and  survival  rate,  a  group  whose  estimates  were 
markedly  inferior. 

Attention  to  correct  answers.  After  making  their  initial 
estimates,  subjects  were  given  the  statistical  answers.  In  an 
effort  to  encourage  them  to  attend  to  those  answers,  they  were 


Table  8 


Contrast  Between  Original  Estimates 
and  Recall  of  True  Values 
Ten  Items  of  Experiment  2 


Death 

Rate 

Number 

Died 

Survival 

Rate 

Number 

Survived 

Median 

Error  Ratio 

Across  Items 

Experiment 

1 

Initial 

.651 

.  380 

.059 

4.66 

Experiment 

2 

Initial 

.  391 

.  369 

.158 

5.82 

Experiment 

2 

Recall 

.415 

.516 

.656 

2.35 

Median 

Error  Factor 

Across  Items 

Experiment 

1 

Initial 

3.24 

2.64 

19.60 

4.66 

Experiment 

2 

Initial 

3.04 

2.90 

10.52 

5.82 

Experiment 

2 

Recall 

2.25 

2.59 

2.64 

2.41 

Percentage  of 

Cases 

Recall  =  True 

Value 

19.3 

15.3 

21.3 

10.4 

Recall  =  Orig. 

Estimate 

4.1 

4.4 

3.0 

5.2 

25 


asked  to  score  their  own  estimates  as  being  too  high  or  too 
low.  One  sign  of  the  attention  they  paid  is  the  accuracy  of 
that  scoring.  There  were  47  errors  in  1480  scoring  opportun¬ 
ities  (  =  3.2%),  approximately  equally  divided  between  incor¬ 
rectly  marking  too  high  or  too  low  (26  vs.  21).  Although  almost 
half  of  the  errors  (21)  were  made  in  the  survival  rate  group, 
this  reflected  primarily  two  subjects  who  generally  scored 
themselves  incorrectly.  Roughly  equal  proportions  of  subjects 
made  at  least  one  error  in  each  group  (overall  percentage  = 

18.2).  These  results  seem  to  indicate  quite  a  high  level  of 
attention. 

Recall  estimates.  Table  7  presents  the  results  of  subjects' 
efforts  to  recall  the  correct  answer  that  they  had  been  given. 

For  the  death  rate  and  number  died  groups,  these  memories  were 
quite  similar  to  subjects'  initial  estimates.  The  respective 
rank  correlations  were  .77  and  .96,  and  few  of  the  paired 
estimates  were  different  by  as  much  as  a  factor  of  two.  Recalled 
estimates  tended  to  be  a  bit  more  accurate  than  the  initial  esti¬ 
mates,  indicating  some  learning  on  subjects'  part.  As  the  bottom 
of  Table  3  indicates,  these  subjects  remembered  the  true  value 
as  given  about  1/6  of  the  time.  Overall,  their  memories  were 
closer  to  the  true  value  than  were  their  initial  estimates  about  2/3 
of  the  time. 

Changes  were,  however,  somewhat  greater  with  the  number 
survived  group  and  much  greater  with  the  survival  rate  group. 

The  rank  correlations  between  initial  and  recall  estimates  were 
.73  (number  survived)  and  .64  (survival  rate).  More  impressivef 
their  error  factors  were  indistinguishable  from  those  of  the 
death  rate  and  number  died  groups.  The  survival  rate  group  re¬ 
membered  the  largest  percentage  of  the  true  answers  (although 
this  percentage  was  only  nominally  greater  than  that  of  the 


2  5 


T  . assai 


death  rate  group,  which  showed  much  less  improvement.  There 
was,  however,  still  considerable  disagreement  among  survival 
rate  subjects  on  the  recall  task,  as  reflected  in  the  low  coef¬ 
ficient  of  concordance.  One  possible  explanation  is  that  ,.he 
group’s  improvement  was  due  primarily  to  shifts  in  the  responses 
of  a  portion  of  subjects,  whose  responses  then  became  discordant 
with  those  of  subjects  who  were  not  responsive  to  the  information 
provided . 

As  Table  8  indicates,  subjects  remembered  the  true  value 
that  they  had  been  told  only  16.6%  of  the  time.  For  individual 
maladies,  the  memory  rate  varied  from  0.6%  (emphysema)  and  1.7% 
(tuberculosis)  to  36.1%  (influenza)  and  48.3%  (pregnancy) . 

These  rates  were  consistent  with  a  serial  position  effect;  the 
two  best- remembered  were  first  and  last  in  the  list,  respectively; 
the  two  worst- remembered  were  fourth  and  sixth,  respectively. 

It  seems  reasonable,  however,  that  personal  relevance  had  some 
contribution  to  memorability.  For  example,  cancer  had  the  third 
best  memory  rate  (22.4%)  despite  being  fifth  in  the  list,  sand¬ 
wiched  between  emphysema  and  tuberculosis.  The  low  rate  of 
subjects  remembering  their  own  original  responses  (about  4%) 
suggests  that  those  numbers  were  not  very  well  ingrained  in 
subjects'  minds.  Whatever  subjects  remembered  when  they  were 
unable  to  recall  the  true  value,  it  was  not  their  previous 
expression. 

Naturalness  of  phrasing.  Table  9  shows  preferences  for 
different  modes  of  expressing  information  about  the  lethality 
of  infectious  hepatitis.  Those  in  the  top  section  represetn 
subjects  who  performed  only  this  task.  They  received  one  of 
four  different  forms,  with  each  mode  occupying  a  different 
ordinal  position  on  each  form.  There  were  no  appreciable  order 
effects;  hence,  the  forms  are  combined  for  the  87  subjects. 
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Table  9 

Naturalness  of  Phrasing 


Number 

of  Choices  in  Each 

Rank 

Phrasings  Judged 

Rank  1 

Rank  2 

Rank  3 

Rank  4 

Mean  Rank 

Subjects  who 
received  no 
other  tasks 

Death  Rate 

37 

24 

12 

14 

2.03 

Number  Died 

26 

43 

14 

4 

1.95 

Survival  Rate 

13 

10 

33 

31 

2.94 

Number  Survived 

12 

9 

28 

38 

3.06 

Subjects  who 
made  estimates 

Death  Rate 

56 

38 

24 

25 

2.13 

Number  Died 

44 

62 

25 

12 

2.03 

Survival  Rate 

30 

24 

52 

38 

2.68 

Number  Survived 

14 

20 

42 

67 

3.13 

Mean  Rankings  of 

Phrasing  Received 

Death 

Number 

Survival 

Number 

Rate 

Died 

Rate 

Survived 

Death  Rate 

1.73 

2.03 

2.41 

2.30 

Number  Died 

2.22 

2,13 

1.89 

1.88 

Survival  Rate 

2.70 

2.78 

2.50 

2.79 

Number  Survived 

3.35 

3,06 

3.19 

2.91 
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Clearly,  subjects  preferred  the  death  phrasings  to  the  survival 
phrasings  and  did  not  consistently  distinguish  between  number 
and  rate  phrasing. 

The  middle  and  lower  portions  of  the  table  summarize  the 
rankings  of  the  subjects  who  had  previously  completed  the  esti¬ 
mation  and  recall  tasks.  Two  orders  were  used  here,  that  appear¬ 
ing  in  the  table  and:  survival  rate,  number  survived,  death 
rate,  number  died.  Again,  no  order  effects  were  found  and 
results  are  collapsed  over  the  two  orders.  As  the  middle  sec¬ 
tion  of  the  table  indicates,  the  popularity  of  the  phrasings 
was  similar  here  to  what  it  was  for  the  "fresh"  subjects  in 
the  top  section.  When  mean  rankings  are  examined  as  a  function 
of  the  phrasing  subjects  had  faced  in  their  estimation  and  re¬ 
call  tasks  (bottom  of  table) ,  much  the  same  pattern  is  seen. 
However,  having  used  a  phrasing  in  a  task  did  tend  to  increase 
its  popularity.  Overall,  mean  rankings  decreased  by  an  average 
of  0.24  for  subjects  who  had  used  a  phrasing.  These  changes 
were  greater  in  the  less  popular  groups.  Thus,  although  judg¬ 
ments  of  naturalness  seem  quite  robust,  tb^y  can  be  affected 
somewhat  by  immediate  experience. 

Discussion 


The  initial  estimates  of  Experiment  2  strongly  replicated 
the  basic  pattern  of  results  from  Experiment  1.  These  four 
formally  equivalent  response  modes  produce  systematically 
varying  subjective  estimates.  Differences  of  a  factor  of  25 
in  the  estimated  lethality  of  a  given  malady  were  not  uncommon. 
Depending  upon  the  method  used,  one  could  find  that  people  tend 
to  systematically  overestimate  (e.g.,  number  survived)  or 
underestimate  (e.g.,  survival  rate)  lethality. 
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Despite  this  divergence  in  the  absolute  accuracy  of  the 
estimates  derived  with  the  different  methods,  the  relative 
lethality  of  the  different  maladies  was  seen  as  quite  similar 
across  methods.  Rank  correlations  between  the  estimates  of  the 
different  groups  were  on  the  order  of  .75.  Moreover,  there  were 
fairly  consistent  tendencies  to  underestimate  or  overestimate 
the  lethality  of  particular  maladies,  relative  to  the  overall 
bias  of  their  group. 

The  similarities  of  the  responses  in  the  two  experiments 
means  that  the  status  of  the  various  explanations  offered  for 
the  results  of  Experiment  1  remain  intact.  The  disagreement 
between  the  survival  rate  and  number  survived  groups'  estimates 
means  that  availability  explanations  are  inadequate.  The  disagree¬ 
ment  between  the  survival  rate  and  death  rate  groups'  estimates, 
despite  their  being  required  to  produce  numbers  with  a  similar 
order  of  magnitude/ means  that  a  predisposition  not  to  use  large 
numbers  cannot  be  the  single  effective  element.  The  lack  of 
any  special  affinity  between  the  estimates  of  subjects  with 
"rate"  and  "number"  tasks  reduces  the  likelihood  of  that  aspect 
being  an  explanatory  factor.  As  noted  in  Footnote  1,  anchoring 
and  adjustment  continues  to  provide  no  predictive  guide.  Thus, 
we  are  left  with  the  same  unsatisfying  (even  if  true)  item- 
specific  or  hybrid  explanations  offered  earlier. 

The  recall  and  naturalness  judgment  tasks  do,  however,  cast 
some  additional  light.  The  two  "death"  perspectives  were  judged 
to  be  more  natural  ways  of  considering  lethality;  moreover, 
they  produced  somewhat  more  accurate  assessments  in  both  experi¬ 
ments.  Despite  being  judged  less  natural  and  producing  poorer 
estimates,  the  "survival"  groups  were  equally  or  more  responsive 
to  information  presented  in  their  perspective.  The  accuracy 
of  their  recalled  estimates  was  indistinguishable  from  that  of 
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the  other  groups.  Particularly  striking  was  the  improvement 
in  the  survival  rate  group.  That  perspective  received  poor 
naturalness  rankings  and  produced  poor  estimates,  yet  subjects 
remembered  20%  of  the  true  answers  presented  in  that  perspec¬ 
tive  . 


Apparently,  perspectives  that  are  not  used  "naturally"  may 
still  be  quite  usable.  As  a  result,  one  might  use  quite  different 
criteria  for  choosing  a  method  by  which  to  assess  people's  current 
store  of  knowledge  and  by  which  to  increase  that  store.  On  the 
basis  of  what  we  have  learned  here,  one  might  reasonably  ask 
for  direct  estimates  of  lethality  to  find  out  what  people  know, 
but  give  them  new  information  in  the  form  of  survival  rates. 

Given  the  weakness  of  our  theoretical  understanding  of  these 
processes,  there  may  be  no  alternative  to  systematic  experimenta¬ 
tion  whenever  one  must  choose  between  formally  equivalent  ways 
of  expressing  a  problem. 

An  unanswered  question  at  this  moment  is  the  extent  to 
which  information  presented  in  one  mode  becomes  accessible  in 
another.  For  example,  how  much  will  teaching  people  survival 
rates  improve  their  direct  estimates  of  the  lethality  rate? 

The  critical  theoretical  question  here  is  whether  people  have 
a  single  coherent  mental  representation  of  lethality  information. 
If  they  do,  then  anything  that  people  learn  becomes,  in  principle, 
accessible  to  whatever  mode  of  questioning  is  employed.  When¬ 
ever  and  however  a  question  is  asked,  people  access  that  bloc 
of  knowledge,  performing  the  inferences  and  transformations 
needed  to  produce  the  desired  response.  It  is  these  inference 
and  transformation  processes  that  lead  to  discrepant  expressions 
of  the  same  knowledge  produced  with  differing  response 
modes.  Thus,  for  example,  the  low  coefficient  of  concordance 
with  the  survival  rate  group  might  mean  that  that  was  indeed  a 
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less  natural  perspective,  requiring  a  more  cumbersome  and 
unreliable  translation  process  to  produce  responses. 

An  alternative  assumption  is  that  people  have  no  such  co¬ 
herent  core  of  knowledge.  Rather,  they  know  somewhat  different 
things  about  death  rates,  survival  rates,  numbers  died,  and 
numbers  survived.  These  four  bodies  of  knowledge  are  sufficiently 
linked  to  produce  similar  rankings  of  the  lethality  of  hazards 
with  greatly  varying  lethality  rates.  However,  they  are  suffi¬ 
ciently  independent  to  allow  for  fairly  discrepant  judgments. 

In  this  light,  the  responses  of  the  four  groups  tell  four  dif¬ 
ferent  stories,  and  the  disagreement  among  subjects'  responses 
reflects  actual  differences  in  their  knowledge,  not  noise  intro¬ 
duced  by  inference  or  processing.  One  might  then  expect  to  find 
the  greatest  disagreement  among  subjects  associated  with  the 
least  accuracy,  suggesting  a  form  of  information  that  people 
do  not  track  very  well  (or  regularly) ;  the  survival  rate  perspec¬ 
tive  would  fit  this  description  quite  well. 

The  question  of  whether  people  have  a  single  store  or  mul¬ 
tiple  stores  of  information  is  hard  to  answer,  even  when  respon¬ 
dents  are  presented  with  concentrated  doses  of  well-characterized 
information  about  completely  novel  tasks  for  which  they  must 
create  storage  schemes  (e.g.,  Estes,  1976;  Hintzman,  1977). 

With  naturally  occurring  information,  such  as  that  dealing  with 
lethality,  a  definitive  answer  is  probably  impossible.  Yet, 
some  idea  of  the  coherence  of  people's  knowledge  on  a  topic  is 
needed  to  enable  one  to  understand  what  they  can  be  expected 
to  learn  from  the  information  with  which  they  are  presented  and 
whether  special  instruction  is  needed  to  help  them  to  integrate 
their  knowledge  and  derive  logically  implied  conclusions. 


32 


FOOTNOTE 


1.  Another  response-bias  explanation,  which  may  be  invoked  for 
any  numerical  estimation  tasks  is  anchoring  and  adjustment. 

The  judge  picks  some  initially  relevant  number  as  a  starting 
point  (or  anchor)  and  then  adjusts  it  to  accommodate  additional 
information.  Although  anchoring  can  be  a  powerful  heuristic 
(Tversky  &  Kahneman,  19' 4),  its  application  in  the  present  tasks 
is  unclear.  For  example,  was  the  number  died  group  anchored 
on  the  total  number  of  deaths,  the  number  of  deaths  per  100,000 
people  in  the  U.S.,  the  number  of  survivors,  the  number  of 
deaths  from  accidents  or  the  violent  causes  (all  of  which  appeared 
on  their  form)  or  some  other  number (s)  of  their  own  creation? 

In  the  absence  of  an  experimental  design  controlling  for  various 
possible  anchors,  all  one  may  be  able  to  do  ~s  to  assume  that 
people  are  using  the  anchoring  heuristic  and  work  backwards  to 
identify  the  anchor (s)  capable  of  producing  the  observed  results. 
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